Systems and methods for note content extraction and management by segmenting notes

ABSTRACT

Techniques for creating and manipulating software notes representative of physical notes are described. A note management system comprises a sensor configured to capture an image data of a physical note, wherein the note is separated into one or more segments using marks, wherein each of the segments is marked by at least one of the marks. The note management system further comprises a note recognition module coupled to the sensor, the note recognition module configured to receive the captured image data and identify the marks on the note, and a note extraction module configured to determine general boundaries of the one or more segments within the captured image data based on the identified marks and extract content using the general boundaries, the content comprises content pieces, each of the content pieces corresponding to one of the one or more segments of the note.

This application claims the benefit of U.S. Provisional ApplicationsNos. 61/844,140, 61/844,152 and 61/844,176, filed Jul. 9, 2013, theentire content of each being incorporated herein by reference.

TECHNICAL FIELD

The present disclosure relates to note content capturing, recognition,extraction, and/or management tools and systems.

BACKGROUND

Physical notes have been broadly used in recording, sharing, andcommunicating ideas and information. For example, during a collaborationsession (e.g., brainstorming session), participants write down ideas onPost-It® notes, whiteboard, or paper, and then share with one another.In addition, people commonly use notes throughout the day to memorializeinformation or content which the individual does not want to forget. Asadditional examples, people frequently use notes as reminders of actionsor events to take in the future, such as to make a telephone call,revise a document or to fill out a time sheet.

For example, in many situations people participate in a collaborationsession by writing information on paper-based notes, such as Post-It®notes. Paper Post-It® notes can readily be removed from a dispenser padof sticky-back paper Post-It® notes and applied to various surfaces,such as whiteboards, documents, the tops of desks, telephones, or thelike. Information can be written on paper Post-It® notes either beforeor after the paper Post-It® notes are detached from their dispenser pador attached to their target surfaces. Paper Post-It® notes can be easilymoved from one surface to another, such as between documents or betweendocuments and the tops of desks, they can overlap edges or boundaries ofdocuments, they can be layered, and they can be moved with the objectsto which they are attached.

Software programs currently exist which permit computer users to createa software-based note in a digital form and to utilize the digital notewithin a computing environment. For example, a computer user may createa digital note and “attach” the digital note to an electronic document adesktop or electronic workspace presented by the computing environment.

SUMMARY

In general, the disclosure describes techniques for creating andmanipulating software notes representative of physical notes.

In one example, a method comprises capturing, by a sensor, image datacomprising a visual representation of a note, wherein the note comprisesa physical note having one or more segments, each of the segments havingat least one mark thereon. The method further comprises identifying, bythe one or more processors, the marks on the note and, based on theidentified marks, determining, by the one or more processors, generalboundaries within the image data for the one or more segments. Themethod comprises extracting, by the one or more processors and from theimage data, content comprising a set of content pieces using the generalboundaries, each of the content pieces corresponding to a different oneof the one or more segments of the note.

In another example, a note management system comprises a sensorconfigured to capture an image data of a physical note, wherein the noteis separated into one or more segments using marks, wherein each of thesegments is marked by at least one of the marks. The note managementsystem further comprises a note recognition module coupled to thesensor, the note recognition module configured to receive the capturedimage data and identify the marks on the note, and a note extractionmodule configured to determine general boundaries of the one or moresegments within the captured image data based on the identified marksand extract content using the general boundaries, the content comprisescontent pieces, each of the content pieces corresponding to one of theone or more segments of the note.

In another example, a non-transitory computer-readable storage devicecomprises instructions that cause a programmable processor to receiveimage data containing a visual representation of a scene having aplurality of physical notes, each of the plurality of physical noteshaving a plurality of segments, each of the segments having at least onemark having a known characteristic. The instructions cause the processorto process the image data to identify the marks within the image dataand determine, based on the known characteristics of the marks, aboundary within the image data of a first physical note of the pluralityof physical notes, and to process the image data to determine, based onthe known characteristic of the marks and the boundary determined forthe physical note, boundaries for each of the segments of the firstphysical note. The instruction cause the processor to extract, from theimage data, respective content for each of the segments of the firstphysical note using and the boundaries determined for each of thesegments of the physical note, create a digital note representative ofthe first physical note, the digital note having data indicating each ofthe segments for the physical note, and associate the respective contentpieces extracted from the image data with the corresponding segments ofthe digital note.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings are incorporated in and constitute a part ofthis specification and, together with the description, explain theadvantages and principles of the invention.

FIG. 1 is a representation illustrating one example of a user capturingan image of a workspace with notes using an image capture device on amobile device.

FIG. 2 is a block diagram illustrating one example of a mobile device.

FIG. 3 is a block diagram illustrating one example of a user applicationto process the input image.

FIG. 4A illustrates an embodiment of a note recognition system.

FIG. 4B illustrates an embodiment of a note management system.

FIGS. 5A-5D illustrate some examples of notes having marks.

FIGS. 5E-5F illustrate some examples of notes with segments.

FIG. 5G illustrates an example of creating segments of a note usingmarks.

FIG. 5H illustrates an example of marks.

FIG. 6A illustrates a flow diagram of an embodiment of a noterecognition and/or management system.

FIG. 6B illustrates a flow diagram of an embodiment of extractingcontent of notes.

FIG. 6C illustrates a flow diagram of an embodiment of authenticating anote.

FIG. 6D illustrates a functional flow diagram of an embodiment ofextracting content of notes or note segments using color classificationalgorithm.

FIG. 6E illustrates a functional flow diagram of an example ofextracting content of notes or note segments having three differentcolors using color classification algorithm.

FIG. 6F illustrates a functional flow diagram of an embodiment ofextracting a set of content from a note with segments.

FIG. 6G illustrations a functional flow diagram of an embodiment ofsegmenting a note and extracting a set of content for the segments.

FIGS. 7A-7D illustrate an example of content extraction process of anote with a mark.

FIGS. 8A-8D illustrate an embodiment of content extraction of aplurality of notes with retroreflective marks.

FIG. 9 illustrates a module diagram of an embodiment of a noterecognition system.

FIG. 10A illustrates a flow diagram of an embodiment of a notemanagement system.

FIG. 10B illustrates examples of how a system may label a note.

FIG. 11 illustrates a module diagram of an embodiment of a notemanagement system.

FIG. 12A illustrates an example of content extraction process of asegmented note with marks.

FIG. 12B illustrates an example of content extraction process of a notewith color segments.

FIG. 12C illustrates an exemplary scenario of marking a white board toextract segmented content.

FIGS. 13A and 13B illustrate an example of extracting colored notesusing color classification algorithm.

FIG. 14 illustrates a graph of classification values vs. colors.

DETAILED DESCRIPTION

The present disclosure describes techniques for creating andmanipulating software notes representative of physical notes. Forexample, techniques are described for recognizing physical notes presentwithin a physical environment, capturing information therefrom andcreating corresponding digital representations of the physical notes,referred to herein as digital notes or software-based notes. Further, atleast some aspects of the present disclosure are directed to techniquesfor managing multiple notes, such as storing, retrieving, editing thedigital notes, categorizing and grouping the digital notes, or the like.

Notes have been broadly used in recording, sharing, and communicatingideas and information. For example, during a collaboration session(e.g., brainstorming session), participants write down ideas on Post-It™notes, whiteboard, or paper, and then share with one another. At leastsome aspects of the present disclosure are directed to systems andmethods of capturing multiple notes and extracting content of notes. Insome embodiments, at least some of the notes include a mark tofacilitate the recognition and extraction of the notes. In some cases,the notes are captured and identified/recognized in one visualrepresentation of a scene. A visual representation of a scene, forexample, can be a digital photograph, or still frame from video ofnote(s) and the surrounding environment. Further, at least some aspectsof the present disclosure are directed to systems and methods ofmanaging multiple notes, such as storing and retrieving the notes,categorizing and grouping the notes, or the like. In some cases, thenote management system can improve the efficiency in capturing andextracting note content from a large number of notes. In addition, thenote management system can improve the efficiency in grouping andmanaging notes.

In general, notes can include physical notes and digital notes. Physicalnotes generally refer to objects with a general boundary andrecognizable content. Physical notes can include the resulting objectsafter people write, draw, or enter via other type of inputs on theobjects, for example, paper, white board, or other objects accepting theinputs. By way of examples, physical notes can include hand-writtenPost-It™ notes, paper, or film, white-board with drawings, posters, andsigns. In some cases, physical notes can be generated using digitalmeans, e.g. printing onto printable Post-It™ notes or printed document.In some cases, one object can include several notes. For example,several ideas can be written on a piece of poster paper or awhite-board. In some implementations, to facilitate the recognition ofthese notes, marks, such as lines, shapes, colors, symbols, markers, orstickers, can be applied to the edges of the notes. Physical notes canbe two-dimensional or three dimensional. Physical notes can have variousshapes and sizes. For example, a physical note may be a 7.62×7.62 cm(3×3 inches) note; a physical note may be a 66.04×99.06 cm (26×39inches) poster; and a physical note may be a triangular metal sign. Insome cases, physical notes have known shapes and/or sizes that conformto standards, such as legal, A3, A4, and other size standards, and knownshapes, which may not be limited to geometric shapes, such as stars,circles, rectangles, or the like. In other cases, physical notes mayhave non-standardized sizes and/or irregular shapes. Digital notesgenerally refer to digital objects with information and/or ideas.Digital notes can be generated using digital inputs. Digital inputs caninclude, for example, keyboards, touch screens, digital cameras, digitalrecording devices, stylus, digital pens, or the like. In some cases,digital notes may be representative of physical notes.

In some cases, notes are used in a collaboration space. Collaborationspace generally refers to a gathering area allowing more than one personto share ideas and thoughts with each other. The collaboration space caninclude virtual spaces allowing a group of persons to share ideas andthoughts remotely, besides the gathering area.

FIG. 1 illustrates an example of a note recognition environment 10. Inthe example of FIG. 1, environment 10 includes a mobile device 15 tocapture and recognize one of more notes 22 from a workspace 20. Asdescribed herein, mobile device provides an execution environment forone or more software applications that, as described, can efficientlycapture and extract note content from a large number of physical notes,such as the collection of notes 22 from workspace 20. In this example,notes 22 may be the results of a collaborative brainstorming sessionhaving multiple participants. As described, mobile device 15 and thesoftware executing thereon may perform a variety of note-relatedoperations, including automated creation of digital notes representativeof physical notes 22 of workspace 20.

In the example implementation, mobile device 15 includes, among othercomponents, an image capture device 18 and a presentation device 28. Inaddition, although not shown in FIG. 1, mobile device 15 may include oneor more processors, microprocessors, internal memory and/or data storageand other electronic circuitry for executing software or firmware toprovide the functionality described herein.

In general, image capture device 18 is a camera or other componentconfigured to capture image data representative of workspace 20 andnotes 22 positioned therein. In other words, the image data captures avisual representation of an environment, such as workspace 20, having aplurality of physical notes. Although discussed as a camera of mobiledevice 15, image capture device 18 may comprise other components capableof capturing image data, such as a video recorder, an infrared camera, aCCD (Charge Coupled Device) array, a laser scanner, or the like.Moreover, the captured image data can include at least one of an image,a video, a sequence of images (i.e., multiple images taken within a timeperiod and/or with an order), a collection of images, or the like, andthe term input image is used herein to refer to the various exampletypes of image data.

Presentation device 28 may include, but not limited to, anelectronically addressable display, such as a liquid crystal display(LCD) or other type of display device for use with mobile device 28. Insome implementations, mobile device 15 generates the content to displayon presentation device 28 for the notes in a variety of formats, forexample, a list, grouped in rows and/or column, a flow diagram, or thelike. Mobile device 15 may, in some cases, communicate displayinformation for presentation by other devices, such as a tabletcomputer, a projector, an electronic billboard or other external device.

As described herein, mobile device 15, and the software executingthereon, provide a platform for creating and manipulating digital notesrepresentative of physical notes 22. For example, in general, mobiledevice 15 is configured to process image data produced by image capturedevice 18 to detect and recognize at least one of physical notes 22positioned within workspace 20. In some examples, the mobile device 15is configured to recognize note(s) by determining the general boundaryof the note(s). After a note is recognized, mobile device 15 extractsthe content of at least one of the one or more notes, where the contentis the visual information of note 22.

As further described below, mobile device 15 may implement techniquesfor automated detection and recognition of physical notes 22 andextraction of information, content or other characteristics associatedwith each of the physical notes. For example, mobile device 15 may allowuser 26 fine grain control over techniques used by mobile device 15 todetect and recognize physical notes 22. As one example, mobile device 15may allow user 26 to select between marker-based detection techniques inwhich one or more of notes 22 includes a physical fiducial mark on thesurface of the note or non-marker based techniques in which no fiducialmark is used.

In addition, mobile device 15 provides user 26 with an improvedelectronic environment for generating and manipulating correspondingdigital notes representative of physical notes 22. As another example,mobile device 15 may provide mechanisms allowing user 26 to easily adddigital notes to and/or delete digital notes from a set of digital notesrepresentative of the brainstorming activity associated with workspace20. In some example implementations, mobile device 15 providesfunctionality by which user 26 is able to record and managerelationships between groups of notes 22.

In some example implementations, mobile device 15 provides functionalityby which user 26 is able to export the digital notes to other systems,such as cloud-based repositories (e.g., cloud server 12) or othercomputing devices (e.g., computer system 14 or mobile device 16).

In the example of FIG. 1, mobile device 15 is illustrated as a mobilephone. However, in other examples, mobile device 15 may be a tabletcomputer, a personal digital assistant (PDA), a laptop computer, a mediaplayer, an e-book reader, a wearable computing device (e.g., a watch,eyewear, a glove), or any other type of mobile or non-mobile computingdevice suitable for performing the techniques described herein.

FIG. 2 illustrates a block diagram illustrating an example of a mobiledevice that operates in accordance with the techniques described herein.For purposes of example, the mobile device of FIG. 2 will be describedwith respect to mobile device 15 of FIG. 1

In this example, mobile device 15 includes various hardware componentsthat provide core functionality for operation of the device. Forexample, mobile device 15 includes one or more programmable processors70 configured to operate according to executable instructions (i.e.,program code), typically stored in a computer-readable medium or datastorage 68 such as static, random-access memory (SRAM) device or Flashmemory device. I/O 76 may include one or more devices, such as akeyboard, camera button, power button, volume button, home button, backbutton, menu button, or presentation device 28 as described in FIG. 1.Transmitter 72 and receiver 74 provide wireless communication with otherdevices, such as cloud server 12, computer system 14, or other mobiledevice 16 as described in FIG. 1, via a wireless communication interfaceas described in FIG. 1, such as but not limited to high-frequency radiofrequency (RF) signals. Mobile device 15 may include additional discretedigital logic or analog circuitry not shown in FIG. 2.

In general, operating system 64 executes on processor 70 and provides anoperating environment for one or more user applications 77 (commonlyreferred to “apps”), including note management application 78. Userapplications 77 may, for example, comprise executable program codestored in computer-readable storage device (e.g., data storage 68) forexecution by processor 70. As other examples, user applications 77 maycomprise firmware or, in some examples, may be implemented in discretelogic.

In operation, mobile device 15 receives input image data and processesthe input image data in accordance with the techniques described herein.For example, image capture device 18 may capture an input image of anenvironment having a plurality of notes, such as workspace 20 of FIG. 1having of notes 22. As another example, mobile device 15 may receiveimage data from external sources, such as cloud server 15, computersystem 14 or mobile device 16, via receiver 74. In general, mobiledevice 15 stores the image data in data storage 68 for access andprocessing by note management application 78 and/or other userapplications 77.

As shown in FIG. 2, user applications 77 may invoke kernel functions ofoperating system 64 to output a graphical user interface (GUI) 79 forpresenting information to a user of mobile device. As further describedbelow, note management application 78 may construct and control GUI 79to provide an improved electronic environment for generating andmanipulating corresponding digital notes representative of physicalnotes 22. For example, note management application 78 may construct GUI79 to include a mechanism that allows user 26 to easily add digitalnotes to and/or deleting digital notes from defined sets of digitalnotes recognized from the image data. In some example implementations,note management application 78 provides functionality by which user 26is able to record and manage relationships between groups of the digitalnotes by way of GUI 79.

FIG. 3 illustrates a block diagram illustrating one exampleimplementation of note management application 78 that operates inaccordance with the techniques described herein. Although described as auser application 77 executing on mobile device 15, the examplesdescribed herein may be implemented on any computing device, such ascloud server 12, computer system 14, or other mobile devices.

In this example, user application 78 includes image processing engine 82that provides image processing and object recognition functionality.Image processing engine 82 may include image communication module 90,note identification module 86 and digital note generation module 88. Inaddition, image processing engine 82 includes image processing APIs 95that provide a library of image manipulation functions, e.g., imagethresholding, masking, filtering, edge detection, and the like, for useby the other components of image processing engine 82.

In general, image data may be stored in storage device 68. In thisexample, note management application 78 stores images 97 within datastorage device 68. Each of images 97 may comprise pixel data forenvironments having a plurality of physical images, such as workspace 20of FIG. 1.

As described herein, note identification module 86 processes images 97and identifies (i.e., recognizes) the plurality of physical notes in theimages. The input image may be processed by note identification module86 using marker and/or non-marker detection processes. Digital notegeneration module 88 generates digital notes 99 corresponding to thephysical notes recognized within the image 97. For example, each ofdigital notes 99 corresponds to one of the physical notes identified inan input image 97. During this process, digital note generation modulemay update database 94 to include a record of the digital note, and maystore within the database information (e.g., content) extracted from theinput image within boundaries determined for the physical note asdetected by note identification module 86. Moreover, digital notegeneration module 88 may store within database 94 metadata associatingthe digital notes into one or more groups of digital notes.

Communication module 90 controls communication of image data betweenmobile device 15 and external devices, such as cloud server 12, computersystem 14, mobile device 16, or image capture device 18. In someexamples, communication modules 90 may, for example, allow a user tocommunicate processed or unprocessed images 97 of environments and/ordigital notes and associated information extracted therefrom includingmetadata from database 68. In some examples, image communication module90 exports this data to a zip file that may be communicated by FTP,HTTP, email, Bluetooth or other mechanism.

In the example of FIG. 1, note management application 78 includes userinterface 98 that constructs and controls GUI 79. As described below,user interface 98 may, in some examples, output for display an inputimage 97 overlaid with the plurality of digital notes 99, where each ofthe digital notes is overlaid in place of a corresponding physical note.In addition, user interface 98 may display a group of digital notes 99that has been designated by the user. This group of digital notes 99 maybe, for example, a subset of the digital notes recognized in aparticular input image 97. User interface 98 may display this designatedgroup (set) of the digital notes on a second portion of GUI 79 and allowuser 26 to easily add or remove digital notes 99 from the designatedgroup.

In some example implementations, user interface 98 provides an imageeditor 96 that allow user 26 to edit the overlay image and/or thedigital notes. In another example, digital note generation module 88 mayinclude a process or processes that enhance the extracted informationfrom the input image.

Additional example details of note management application 78 fordetecting and recognizing physical notes are described in U.S. PatentApplication 61/844,140, filed July 9, entitled SYSTEMS AND METHODS FORNOTE RECOGNITION AND MANAGEMENT USING COLOR CLASSIFICATION,” U.S. PatentApplication 61/844,152, filed Jul. 9, 2013, entitled “SYSTEMS ANDMETHODS FOR NOTE CONTENT EXTRACTION AND MANAGEMENT USING SEGMENTEDNOTES, and U.S. Patent Application 61/844,176, filed Jul. 9, 2013,“SYSTEMS AND METHODS FOR NOTE CONTENT EXTRACTION AND MANAGEMENT BYSEGMENTING NOTES,” the entire contents of each of which are incorporatedherein by reference.

To better understand the present disclosure, FIG. 4A illustrates anembodiment of a note recognition system 100A. The system 100A caninclude a processing unit 110, one or more notes 120, a sensor 130, andnote content repository 140. The processing unit 110 can include one ormore processors, microprocessors, computers, servers, and othercomputing devices. The sensor 130, for example, an image sensor, isconfigured to capture a visual representation of a scene having the oneor more notes 120. The sensor 130 can include at least one of a camera,a video recorder, an infrared camera, a CCD (Charge Coupled Device)array, a scanner, or the like. The visual representation can include atleast one of an image, a video, a sequence of images (i.e., multipleimages taken within a time period and/or with a order), a collection ofimages, or the like. The processing unit 110 is coupled to the sensor130 and configured to receive the visual representation. In some cases,the processing unit 110 is electronically coupled to the sensor 130. Theprocessing unit 110 is configured to recognize at least one of the oneor more notes 120 from the visual representation. In some embodiments,the processing unit 110 is configured to recognize note(s) bydetermining the general boundary of the note(s). After a note isrecognized, the processing unit 110 extracts the content of the note. Insome cases, the processing unit 110 is configured to recognize andextract the content of more than one note 120 from a visualrepresentation of a scene having those notes 120.

In some cases, the processing unit 110 can execute software or firmwarestored in non-transitory computer-readable medium to implement variousprocesses (e.g., recognize notes, extract notes, etc.) for the system100A. The note content repository 140 may run on a single computer, aserver, a storage device, a cloud server, or the like. In some othercases, the note content repository 140 may run on a series of networkedcomputers, servers, or devices. In some implementations, the notecontent repository 140 includes tiers of data storage devices includinglocal, regional, and central. The notes 120 can include physical notesarranged orderly or randomly in a collaboration space and the sensor 130generates a visual representation of the notes 120 in the collaborationspace.

In some embodiments, at least some of the one or more notes 120 includea mark, which can facilitate the identification, recognition, and/orauthentication of a note. In some embodiments, a mark includes at leastone of a barcode, a color block, a color code, a fiduciary mark, atrademark logo, a dot, a hole, and the like. The shape and/or color ofthe note itself may be used as the mark to facilitate theidentification, recognition, and/or authentication of a note. In somecases, the mark can include a plurality of elements arranged in certainpatterns, for example, fiduciary marks at four corners of a rectangularnote. In some other cases, the mark can include a plurality of elements,where at least some elements are non-visible elements that can be usedto provide authentication information, for example, RFID (radiofrequency identification) tags. By way of example, a mark can be madeusing at least one of a retroreflective material, an optically variableink, a colored ink, infrared absorbing ink, fluorescent ink, watermark,glossy material, iridescent material, multi-layer optical film,colloidal crystals, perforated marks, structured color, floating image,window thread, or the like. In some embodiments, the processing unit 110first recognizes the mark on a note from a visual representation,determines the location of the mark on the visual representation, andthen extracts the content of the note based on the recognized mark. Insome cases, the processing unit 110 extracts the content of the notebased upon the recognized mark, a known predetermined shape of the note,and/or a known relative position of the mark on the note. In someimplementations, the processing unit 110 extracts the content of thenote from the visual representation in real-time (i.e., process the datain a transitory storage) without storing the visual representation in anon-transitory storage.

In some implementations, the note recognition system 100A can include apresentation device (not shown in FIG. 4A) to show to a user which notesare recognized and/or which notes' content have been extracted. Further,the note recognition system 100A can present the extracted content viathe presentation device. In some embodiments, the processing unit 110can authenticate a note before extracting the content of the note. Ifthe note is authenticated, the content will be extracted and stored inthe note content repository 140. In some cases, the processing unit canextract the authentication information from the mark on the note. Insuch cases, the authentication information can be visible or non-visibleon the mark. For example, the mark on a note can include a symbol, forexample, a company logo, a matrix code, a barcode, a color code, or thelike. As another example, the mark on the note can include a tag tostore information that can be retrieved by a suitable reader. Forexample, the mark can include a RFID tag, a near field communication(NFC) tag, or the like.

In some embodiments, the sensor 130 can generate a first visualrepresentation of a scene having several notes 120, for example, takinga photograph or a video clip of the notes with the surroundingenvironment. Each of the notes has a mark. The processing unit 110identifies the marks, determines the location of the marks, and uses thelocation of the marks to control the sensor 130 to generate a secondvisual representation of the scene having the notes 120, for example,taking a zoomed-in image of the notes. The processing unit 110 canfurther recognize and extract content of notes from the second visualrepresentation.

FIG. 4B illustrates an embodiment of a note management system 100B. Inthis embodiment, the note management system 100B includes processingunit 110, one or more notes 120, one or more note sources 150, and anote content repository 140. In some cases, the system 100B includes apresentation device 160. The processing unit 110, the notes 120, and thenote content repository 140 are similar to the components for the noterecognition system 100A as illustrated in FIG. 4A. The note sources 150can include sources to provide content of physical notes, such as avisual representation of a scene having one or more notes, and sourcesto provide content of digital notes, such as a data stream entered froma keyboard. In some embodiments, the note management system 100Bincludes a first source and a second source, and the first source is avisual representation of a scene having one or more notes 120. The firstsource and the second source are produced by different devices. Thesecond source includes at least one of a text stream, an image, a video,a file, and a data entry. The processing unit 110 recognizes at leastone of the notes from the first source and extracts the content of thenote, as discussed in the note recognition system 100A. In some cases,the processing unit 110 labels the note with a category. The processingunit 110 can label a note based on its specific shape, color, content,and/or other information of the note. For example, each group of notecan have a different color (e.g., red, green, yellow, etc.). In somecases, a note 120 can include mark that has one or more elements and theprocessing unit 110 can label the note based on information extractedfrom the mark.

In some embodiments, the note management system 100B can include one ormore presentation devices 160 to show the content of the notes 120 tothe user. The presentation device 160 can include, but not limited to,an electronically addressable display, such as a liquid crystal display(LCD), a tablet computer, a projector, an electronic billboard, acellular phone, a laptop, or the like. In some implementations, theprocessing unit 110 generates the content to display on the presentationdevice 160 for the notes in a variety of formats, for example, a list,grouped in rows and/or column, a flow diagram, or the like.

Various components of the note recognition system and note managementsystem, such as processing unit, image sensor, and note contentrepository, can communicate via a communication interface. Thecommunication interface includes, but not limited to, any wired orwireless short-range and long-range communication interfaces. Theshort-range communication interfaces may be, for example, local areanetwork (LAN), interfaces conforming to a known communications standard,such as Bluetooth standard, IEEE 802 standards (e.g., IEEE 802.11), aZigBee or similar specification, such as those based on the IEEE802.15.4 standard, or other public or proprietary wireless protocol. Thelong-range communication interfaces may be, for example, wide areanetwork (WAN), cellular network interfaces, satellite communicationinterfaces, etc. The communication interface may be either within aprivate computer network, such as intranet, or on a public computernetwork, such as the internet.

FIGS. 5A-5C illustrate some examples of notes having marks affixedthereon. As illustrated in FIG. 5A, the mark on the note 200A has twoelements, element 210A and element 220A. The elements 210A and 220A canhave different size, shape, relative position and/or materialcomposition. For example, element 210A is a barcode as an identifier forthe note and element 220A is a small rectangle of retro-reflective inkthat can be used to determine the boundary of the note. As illustratedin FIG. 5B, the mark on the note 200B can have four elements 210B, 220B,230B, and 240B. The four elements may have similar or different sizes,shapes, and material compositions. The mark can be used to recognize thesize, location, orientation, distortion, and other characteristics ofthe note, which can be used in content extraction and enhancement. Asillustrated in FIG. 5C, the mark on the note 200C has one element 210C.The note 200C has a non-rectangular shape. In some embodiments, theelement 210C includes an identifier, for example, to identify the typesand the grouping for a note. In some cases, the identifier is a uniqueidentifier.

FIG. 5D illustrates another example of notes having marks. The marks arethe color of the notes. An image 200D capturing twelve notes isillustrated in FIG. 5D. Notes 210D, 212D, and 214D are magenta. Notes220D, 222D, and 224D are yellow. Notes 230D, 232D, and 234D are blue.Notes 240D, 242D, and 244D are orange.

FIGS. 5E and 5F illustrate some examples of notes with segments. FIG. 5Eillustrates a note 200E with four colored segments. Each segment has asolid color. For example, segment 210E is magenta; segment 220E isyellow; segment 230E is blue; and segment 240E is orange. FIG. 5Fillustrates a note 200F with four segments and each segment has acolored boundary. For example, segment 210F has a magenta boundary;segment 220F has a yellow boundary; segment 230F has a blue boundary;and segment 240F has an orange boundary. In some implementations, thecolored boundary has certain line width (e.g., more than 0.1 mm, morethan 2% of the overall width of the segment, etc.) such that theboundary can be recognized by an image processing algorithm. In somecases, the color of the segment or the boundary of the segment can beused to categorize the segment. For example, magenta segments orsegments with magenta boundary may be associated with a category A andyellow segments or segments with yellow-colored boundary may beassociated with a category B. In some other cases, the color of thesegment or the boundary of the segment can be used to denote a status orother functional information regard the note. For example, red maydenote deletion of the note content; and orange may denote note to becompleted.

FIG. 5G illustrates an example of creating segments of a note usingmarks. The note 200G can be any media with content, for example, aprinted document, a white board having writing and/or drawing, a pieceof paper with handwritten notes, or the like. The marks can be generatedusing a pen (i.e., the pen or highlighter 260G illustrated in FIG. 5G),printed marks, flags, stickers, colored tapes, highlighter, coloredpaper, printed flags with handwritten content, or the like. The note200G has four segments, 210G, 220G, 230G, and 240G, where each of thesegments has a rectangular mark. In some implementations, a segment canbe marked at two opposing corners, such as top left corner and bottomright corner. As illustrated in FIG. 5H, the corner marks can be ofpredetermined shapes of known sizes that are affixed to the physicalnote, as shown as 210H and 220H. For example, a user may affix marks210H, 220H to the corners of a physical note or to bracket an areawithin a larger note (e.g., note 200G) where the bracketed region is tobe considered a separate region (segment) from which to extract content.The marks 210H and 220H are elbow-shaped and symmetric. Marks caninclude, but not limited to, one or more of lines, arrows, star-shapedmarks, elbow-shaped marks, rectangular marks, circular marks,ellipse-shaped marks, polygon-shaped marks, and geometric-shaped marks.In some embodiments, marks used for a corner can be in any geometricshapes, for example, rectangle, circle, star, or the like. In addition,marks can have asymmetric elbow-like shapes.

FIG. 6A illustrates a flow diagram of an embodiment of a noterecognition and/or management system, such as note managementapplication 78 of mobile device 15 (FIG. 1) or note recognition system100A (FIG. 4A). Initially, the system captures a visual representationof plurality of notes (step 310A). In some embodiments, the notes arephysical notes and it is more efficient to capture more than one note ata time. Next, the system recognizes one or more of the plurality ofnotes from the visual representation (step 320A). For example, thesystem can recognize a specific mark on a note and subsequentlydetermine the general boundary of the note. The system extracts contentof the one or more of the plurality of notes (step 330A). At this time,the system may create a respective digital note representative of acorresponding one of the recognized physical notes. The system mayassociate the content extracted from the visual representation for agiven physical note with the corresponding digital note representativeof the physical note. In some embodiments, the system can apply imagetransformation to at least part of the visual representation beforeextracting content. In some other embodiments, the system can applyimage enhancement or other image processing technique to improve thequality of the extracted content. In yet other embodiments, the systemcan further recognize text and figures from the extracted content.

FIG. 6B illustrates a flow diagram of an embodiment of extractingcontent of notes. First, a visual representation is received by thesystem (step 310B). The system recognizes a mark on a note from thevisual representation (step 320B). After the position and/or shape ofthe mark is determined, the system may optionally perform geometriccorrection and cropping to the visual representation (step 330B). Basedon the recognized position and/or shape of the mark on the note, thegeneral boundary of the note is recognized on the visual presentation(step 340B). In some embodiments, the system may receive two visualrepresentations of the same set of notes, where each of the notes has amark. The first visual representation is taken using a light sourcesuitable to capture the marks on the notes. For example, the lightsource can be infrared light for marks using infrared sensitive ink. Themarks are recognized in the first visual representation and thepositions of the marks are determined. The second visual representationcan capture the set of notes and the system can determine the generalboundary of each note based on its mark respectively. For example, themarks can be made with retro-reflective materials and the first visualrepresentation can be captured with a separate light source (e.g.,flash, etc.) and the second visual representation can be capturedwithout a separate light source. After the general boundary of a note isdetermined, the system extracts the content of the note (step 350B). Atthis time, the system may create a respective digital noterepresentative of a corresponding one of the recognized physical notes.The system may associate the content extracted from the visualrepresentation for a given physical note with the corresponding digitalnote representative of the physical note. As another example, where thevisual representation is an image, the system may crop the imageaccording to the determined general boundary of the note. Optionally,the system may enhance the content of the note (step 360B), for example,by changing the contrast, brightness, and/or using other imageprocessing techniques (e.g., white balancing, contrast stretching,etc.). In some cases, the system may update the note with a statusindicating its content extracted.

FIG. 6C illustrates a flow diagram of an embodiment of authenticating anote. First, obtain an authentication signal from a note or a stack ofnotes (step 310C). In one embodiment, the authentication information isa visual component (e.g., a hologram) of a mark on a note and theauthentication signal can be obtained by extracting the visual componentfrom a visual representation capturing the note. In another embodiment,the authentication information is contained in an electronic component(e.g., a RFID tag) of a mark on a note and the authentication signal canbe obtained using a suitable reader (e.g., a RFID reader). Next, thesystem verifies the authentication signal (step 320C). The system canstart the note management application or use the note(s) if the note(s)are authenticated (step 330C). In some cases, authentication can be donebefore capturing the notes.

FIG. 6D illustrates a functional flow diagram of an embodiment ofextracting content of notes or note segments using color classificationalgorithms. One or more steps in the flow diagram are optional, forexample, the training steps for computing classification function(s),also referred to as classifier, (step 350D-365D) are not necessary ifthe classification function(s) are provided. Initially, image data ofcapturing a visual representation of note(s) or a note with notesegment(s) (step 310D) within or without the surrounding environment isreceived. In some cases, an image is retrieved from the visualrepresentation (image data) (step 315D) if the visual representation isnot a single image. For example, the visual representation is a set ofimages, and the image retrieved is an aggregation of at least part ofthe set of images. As another example, the visual representation is avideo, and the image retrieved is a combination of several or all framesof the video. Another optional step is to convert the image to adesirable color space (step 317D). The applicable color space includes,but not limited to, RGB (red, green, and blue), LAB (e.g., Hunter 1948L,a,b color space, CIE 1976 (L*, a*, b*) color space), CMYK (cyan,magenta, yellow, and key (black)), HSV (hue, saturation, and value), HSL(hue, saturation, and lightness), HSI (hue, saturation, and intensity),sRGB (standard red, green, and blue) color space. Next, apply one ormore classification functions to color values for each pixel or group ofpixels in the image (step 320D). The classification functions can becomputed using the training steps 350D-365D. The classificationalgorithms can be, for example, linear discriminant analysis, quadraticclassifier, Gaussian Mixture Models, Boosted Decision Trees, SupportVector Machines or the like.

Some classifiers are generative in nature while others arediscriminative. In general, generative classifiers generate anindividual model for each class (in our case a color) and a queriedpixel/group of pixels value is given a probability score as to whetherit belongs to that class or not. Discriminative classifiers on the otherhand model the boundary between two or more classes (2-class andmulticlass classification respectively). Generative models provideeasier generalizability to new classes (colors) that are not alreadymodeled while separate discriminative models have to be retrained toevery existing class (colors). Many examples of generative anddiscriminative classifiers are described in Christopher M. Bishop, 2006,Pattern Recognition and Machine Learning (Information Science andStatistics), Springer-Verlag New York, Inc., Secaucus, N.J., USA, theentire content of which is incorporated herein by reference. Someexamples of generative models are naïve Bayes classifier, GaussianMixture Models and other probabilistic Bayesian models using Gaussian orother priors, and Dictionary Learning, such as those described in MichalAharon, Michael Elad, and Alfred Bruckstein (2006), “K-SVD: An Algorithmfor Designing Overcomplete Dictionaries for Sparse Representation”, IEEETransactions on Signal Processing 54 (11): 4311-4322), the entirecontent of which is incorporated herein by reference. Some examples ofdiscriminative classification models are neural networks, Support VectorMachines, Decision Trees, Linear and Quadratic discriminateclassification, logistic regression. Other example classificationmethods are neither generative nor discriminative e.g. nearest neighborclassification algorithm. In addition, the performance of any of theabove mentioned classification models can be improved by ensemblelearning algorithms that combine multiple instances of classificationmodels mentioned above. Ensemble learning may be implemented as Bagging,Boosting, and their variants.

Using the classification algorithms, indicators indicative of colorclasses for each pixel or group of pixels is generated in the imagebased on color values of the pixel or group of pixels (step 325D). Acolor class includes a particular range of wavelength or can be an“other” color class referring to any other color besides the colorclasses of interest. For example, a color class can be magenta, yellow,blue, orange, etc. An indicator can be represented by, for example, anumber, a code, a symbol, an alphanumerical token, a color value, agrayscale value, or the like.

Based on the indicators, the general boundary of note(s) or notesegment(s) can be determined (step 330D). In one embodiment, theadjacent pixels with a same indicator are grouped into a region and theboundary of the region can be determined. In some cases, the indicatorsare further processed using image processing principles, for example,noise filtering, morphological opening, or the like, before the generalboundary is determined. In some embodiments, the shape and/or the sizeof a note or note segment is predetermined, e.g., predefined at the timethe note was manufactured, which can be used to determine and/or filterthe general boundary of note(s) or note segment(s). At this time, thesystem may create a respective digital note representative for each ofthe physical notes for which a boundary is determined.

Using the general boundaries, the content of the note(s) or notesegment(s) can be extracted (step 335D). The system may associate thecontent extracted from the visual representation for a given physicalnote with the corresponding digital note representative of the physicalnote. In some embodiments, the note(s) or note segment(s) may havecolored boundaries or colored mark(s) at one or more locations insteadof having solid color. In such embodiments, a same color classificationalgorithm can be used to identify the boundaries or marks and furtherextract the content of the note(s) or note segment(s). In some cases, aset of content, where each piece of content corresponds to a note or anote segment, can be presented to user via a graphical interface or in areport. In some implementations, the set of content can be grouped withcategories, where each category is associated with a color, a shape, asize, or a combination thereof used in the note. For example, notes210D, 212D and 214D illustrated in FIG. 5D are associated with acategory A and notes 220D, 222D, and 224D are associated with a categoryB. As another example, a note segment with boundary of magenta (e.g.,note segment 210F illustrated in FIG. 5F) is associated with category Aand a note segment with boundary of yellow (e.g., note segment 220Fillustrated in FIG. 5F) is associated with category B.

In some cases, the generated indicators only indicate whether thepixel's or groups of pixels' color value is a particular color class.For example, an indicator of 1 indicates the pixel or group of pixels isyellow and an indicator of 0 indicates the pixel is not yellow. In suchcases, a set of indicators will be generated for each color class ofinterest. In some embodiment, steps 320D-335D are repeated for eachcolor class of interest, for example, magenta, yellow, blue, and orange.

In some embodiments, the classification functions can be determinedusing the steps illustrated in FIG. 6D. First, a set of training imagesare provided (step 350D). In some cases, the training images can includenotes or note segments with the same color classes of interest. Thetraining images can be captured under one or more lighting conditionsand/or one or more different cameras. For example, the training imagescan be a set of six images captured in three lighting conditions usingtwo different cameras. The training images are optionally converted to adesired color space (step 355D). Train the classifier (step 360D) usingthe known color for corresponding color values of a pixel or group ofpixels, for example using equations (1)-(4) as discussed below. The setof classification function(s) for color class is then computed (step365D). In some cases, a set of classification function(s) for only onecolor class is obtained using steps 350D-365D. These steps are repeatedto obtain classification functions for each of other color classes ofinterest.

In some embodiments, a color class can include two or more similarcolors. The classification function for such a color class can beobtained using the same process as discussed above. In some cases, ahierarchical approach may be used for extracting notes or note segmentshaving more than two colors, as illustrated in FIG. 6E. FIG. 6Eillustrates a functional flow diagram of an example of extractingcontent of notes or note segments having three different colors (yellow,orange, and blue) using color classification algorithm. Initially, thesystem uses a color classification algorithm to determine whether apixel has a color that belongs to a color class including yellow andorange (step 310E). If the color is yellow or orange, the system furtherevaluates whether the color is yellow (step 320E): if it is yellow, anindicator for yellow can be associated with the pixel or group of pixels(step 340E); and if it is not yellow, an indicator for orange can beassociated with the pixel or group of pixels (step 350E). If the colordoes not belong to the color class including yellow and orange, thesystem evaluates whether the color is blue (step 330E) using the colorclassification algorithm: if the color is blue, an indicator for bluecan be associated with the pixel or group of pixels (step 360E); and ifthe color is not blue, an indicator of other color can be associatedwith the pixel or group of pixels (step 370E).

In one embodiment, an image of captured notes or note segments can beconverted into LAB color space. The LAB color space separates differentcolors on the ab plane, and it is possible to differentiate differentcolor in this color space using a threshold approach. Table 1 lists thepseudo code to an example of extracting notes or note segments using athreshold approach.

TABLE 1 Pseudo Code for Simple Color “b” Threshold Load an image (from afile or using other methods) Convert to Lab color space If (b value >75% maximum b value) grayscaleResult = 255 else grayscaleResult = 0Morphological erode grayscaleResult 2 passes Median filtergrayscaleResult Morphological dilate grayscaleResult 2 passes Findcontours of all regions with same grayscaleResult Eliminate smallregions (less than 1% of total image size) Extract notes or notesegments

While the above pseudocode and description explains a two class lineardiscriminate classification process, there are multiclass extensions ofthe same, such as those described in Christopher M. Bishop. 2006.Pattern Recognition and Machine Learning (Information Science andStatistics). Springer-Verlag New York, Inc., Secaucus, N.J., USA.,incorporated herein by reference.

In some cases, the threshold approach may not be adaptive to variouscapture conditions, such as variation of colors, lighting, camera type,ambience, etc. To address this issue, adaptive models can be used, whichcan learn the best representations for different colors that are robustto various capture conditions and additional colored notes. One suchmodel is the Fischer's Linear Discriminant model or Fischer's LinearDiscriminant Classifier, as described by Mika, S.; Ratsch, G.; Weston,J.; Scholkopf, B.; Muller, K., Fisher discriminant analysis withkernels, Neural Networks for Signal Processing IX, 1999, Proceedings ofthe 1999 IEEE Signal Processing Society Workshop., pages 41-48, August1999.

Given a set of m observations X (i.e., the training images), where eachobservation is 3 dimensional in this case (L, a and b values for the LABcolor space) and the corresponding color class for all observations Y, aweight vector w is to be determined such thatw·X+c<0 if Y=1(belongs to the color class of interest), and  (1)w·X+c>=0 if Y=0(does not belong to color class of interest),  (2)where w·X is a dot product operation and c are off-set values. Theweights w depends on the mean values of L, a, and b of each color classand the covariance matrix of each color class. Specifically, the weightw maximizes S, whereS=σ ² _(between-class)/σ² _(within-class)  (3)i.e., w is learned such that the between-class variance is maximized andwithin-class variance is minimized.This results in a w=(Σ_(y=0)+Σ_(y=1))⁻¹(μ_(y=1)−μ_(y=0))  (4)The class mean and covariance (μ and Σ) can be estimated from thetraining image. If it is assumed that the class covariance are identicalthen the model is linear. If this assumption is not made, then the modelis a quadratic discriminant classifier. The classification method forthe quadratic discriminant classifier has an additional term:X ^(T) QX+wX+c<0 if Y=1(belong to class in question)  (5)X ^(T) QX+wX+c>=0 if Y=0(does not belong to class in question)  (6)

For example, after the linear discriminant model is applied to an imagehaving magenta notes and other background color, the resultingprojections are scalar values and the distributions of the scalarprojection values are shown in FIG. 14. The classifier can be furthertrained to recognize color in images captured in various conditionsusing the same process and equations. As an example, for a set ofcoefficients [w1 w2 w3 c] where w1, w2, and w3 are the coefficientscorresponding to L, a, and b respectively and c is the offset, thecoefficients for color class blue is [w1=0.05056 w2=0.34929 w3=0.4381c=−106.71]; and the coefficients for color class yellow is [w1=0.0068796w2=0.027731 w3=−0.14887 c=18.812].

Table 2 lists the pseudo code for an embodiment of using lineardiscriminant classifier and some examples of filtering steps. Here,Lcoefficent, Acoefficient, Bcoefficient and offset correspond to w1, w2,w3 and c in the above example. The example of filtering assumes thenotes are rectangular. Other color spaces such as RGB and HSV can followsimilar approaches, and the pseudo code is listed in Table 3 and Table4.

TABLE 2 Pseudo Code for Linear Discriminant Classifier (LDC) in LABcolor space Load RGB image (from file or other method) Convert to LABcolor space LDC = L * Lcoefficient + A * Acoefficient + B *Bcoefficient + offset If (LDC < 0.0) grayscaleResult = 255 elsegrayscaleResult = 0 Morphological erode grayscaleResult 5 passes Medianfilter grayscaleResult Morphological dilate grayscaleResult 5 passesFind contours of all regions Delete region contours for regions with anarea less than a specified percent of image area Find best fit rotatedrectangle of each remaining region Delete region contours for regionswith an area of less than 0.8 times the area of the best fit rotatedrectangles found in the previous step Extract notes or note segments

A morphological opening process can be used to reduce noise, whichincludes morphological erosion followed by dilation. In some cases, themorphological erosion and dilation can use a structuring element whichis disk shaped or circular. For example, the size of the structuringelement is set at 5 by 5 for an image with a maximum dimension of 800pixels. If the image is larger, the size of the structural element canbe bigger. This type of noise reduction approach are described in ImageAnalysis and Mathematical Morphology by Jean Serra, ISBN 0-12-637240-3(1982); Image Analysis and Mathematical Morphology, Volume 2:Theoretical Advances by Jean Serra, ISBN 0-12-637241-1 (1988); and AnIntroduction to Morphological Image Processing by Edward R. Dougherty,ISBN 0-8194-0845-X (1992).

TABLE 3 Pseudo Code for Linear Discriminant Classifier (LDC) in RGBcolor space Load RGB image (from file or other method) LDC = R *Rcoefficient + G * Gcoefficient + B * Bcoefficient + offset If (LDC <0.0) grayscaleResult = 255 else grayscaleResult = 0 Morphological erodegrayscaleResult 5 passes Median filter grayscaleResult Morphologicaldilate grayscaleResult 5 passes Find contours of all regions Deleteregion contours for regions with an area less than a specified percentof image area Extract notes or note segments

TABLE 4 Pseudo Code for Linear Discriminant Classifier (LDC) in HSVcolor space Load RGB image (from file or other method) Convert to HSVcolor space LDC = H * Hcoefficient + S * Scoefficient + V *Vcoefficient + offset If (LDC < 0.0) grayscaleResult = 255 elsegrayscaleResult = 0 Morphological erode grayscaleResult 5 passes Medianfilter grayscaleResult Morphological dilate grayscaleResult 5 passesFind contours of all regions Delete region contours for regions with anarea less than a specified percent of image area Extract notes or notesegments

FIG. 6F illustrates a functional flow diagram of an embodiment ofextracting a set of content from a note with segments. First, the systemreceives a visual representation of a note having a plurality ofpredefined color segments (step 310F). That is, image data capturing thevisual representation may be received in which the physical note hasbeen manufactured to have a plurality of different color segments. Insome cases, an image from the visual representation (step 320F) may beretrieved, e.g., if the image data comprises video or a collection of aplurality of images. If the visual representation is an image, the imagewill be used for further processing. Next, indicators indicative ofcolor classes of respective pixels or group of pixels are generated inthe image (step 330F), using, for example, a color classificationalgorithm. Using the indicators, general boundaries of the colorsegments are determined (step 340F). Extract a set of content from thenote segments (step 350F). In some cases, each piece of content isextracted from a corresponding note segment. Optionally, present a listof content to a user (step 360F). In some cases, a piece of content canbe associated with a category. A note management system may use thecolor in the color segment to select the category for the piece ofcontent. At this time, the system may create a respective digital noterepresentative of a corresponding one of the recognized physical notes.As one example, the record may be created to contain a plurality ofdatabase fields, including fields for storing the particular content ofeach color segment of the physical note. The system may associate thecontent extracted from the visual representation for a given physicalnote with the corresponding digital note representative of the physicalnote. Moreover, the system may associate the content piece extractedfrom each color segment with the digital representation of each colorsegment. For example, the system may update the fields for therespective record within the database to store respective ones of thecontent pieces for the different color segments.

FIG. 6G illustrations a functional flow diagram of an embodiment ofsegmenting a note and extracting a set of content for the segments.First, a user may segment a note using marks (step 310G). The marks canbe applied to one or more corners, along at least a part of the boundaryof the segment. A note management system can capture an image of thenote (step 320G). The system further identifies the marks in the image(step 330G). The general boundaries of the segments can be determined(step 340G). In some cases, some additional information, such as therelative positions of the marks, can be used to determine the generalboundaries. Extract a set of content from the note segments using thedetermined boundaries (step 350G). In some cases, each piece of contentis extracted from a corresponding note segment. Optionally, present alist of content to a user (step 360G). At this time, the system maycreate a respective digital note representative of a corresponding oneof the recognized physical notes. As one example, the record may becreated to contain a plurality of database fields, including fields forstoring the particular content of each color segment of the physicalnote. The system may associate the content extracted from the visualrepresentation for a given physical note with the corresponding digitalnote representative of the physical note. Moreover, the system mayassociate the content piece extracted from each color segment with thedigital representation of each color segment. For example, the systemmay update the fields for the respective record within the database tostore respective ones of the content pieces for the different colorsegments.

FIGS. 7A-7D illustrate an example of content extraction process of anote with a mark. First, as illustrated in FIG. 7A, a visualrepresentation 400 of a note 410 is captured. The note 410 has a mark420, which can be a barcode, a color code, a matrix code, a color block,or the like. Next, as illustrated in FIG. 7B, the system determines thegeneral boundary 430 of the mark on the visual representation andrecognizes the mark 420. In some cases, the note 410 may be slanted inthe visual representation 400, as illustrated in FIG. 7C. In some othercases, the visual representation 400 may be taken with geometricdistortion. The system may use the determined general boundary of themark 420 or a portion of the mark 420 to determine the necessary imagetransformation and correction to the visual representation 400 to obtainthe note content. FIG. 7D illustrates that the system extracts thecontent of the note 410 after the previous analysis and/or imageprocessing.

FIGS. 8A-8D illustrate an embodiment of content extraction of aplurality of notes with retroreflective marks. A noterecognition/management system receives a visual representation 500A,which captures three notes 510A, and each note 510A has a mark 520having two elements—two retroreflective rectangular tags at upper leftand lower bottom corners. Because of the optical property of theretroreflective material, the mark 520 is substantially brighter thanthe rest of the note. FIG. 8B illustrates the marks 520 are muchbrighter than the notes 510A and the background. In some embodiments,the system may use image processing to transform the visualrepresentation of 500A as illustrated in FIG. 8A to 500B as illustratedin FIG. 8B. In some alternative embodiments, the system may generateanother visual representation 500B of the three notes 510A with adifferent light source, for example, a flashlight. The system can easilyidentify the marks 520 from the visual representation 500B. After themarks 520 are identified, the system can extract the content of thenotes 510C, as illustrated in FIG. 8C. In some cases, the system may useimage processing techniques (e.g., adaptive histogram equalization) toenhance the extracted content 510D, as illustrated in FIG. 8D.

FIG. 9 illustrates a module diagram of an embodiment of a noterecognition system 600. In the embodiment as illustrated, the system 600includes a note capturing module 620, a note recognition module 630, anda note extraction module 640. Various components of the note recognitionsystem 600 can be implemented by one or more computing devices,including but not limited to, circuits, a computer, a processor, aprocessing unit, a microprocessor, and/or a tablet computer. In somecases, various components of the note recognition system 600 can beimplemented on a shared computing device. Alternatively, a component ofthe system 600 can be implemented on multiple computing devices. In someimplementations, various modules and components of the system 600 can beimplemented as software, hardware, firmware, or a combination thereof.In some cases, various components of the note recognition system 600 canbe implemented in software or firmware executed by a computing device.Additionally, various components of the system 600 can communicate viaor be coupled to via a communication interface, for example, a wired orwireless interface. The note capturing module 620 is configured tocapture a visual representation of a plurality of notes. In someembodiments, the note capturing module 620 includes an image sensor. Thenote recognition module 630 is coupled to the note capturing module 620,the note recognition module is configured to receive the captured visualrepresentation and determine a general boundary of one of the pluralityof notes from the captured visual representation. The note extractionmodule 640 is configured to extract content of the one of the pluralityof notes from the captured visual representation based on the determinedgeneral boundary of the one of the plurality of notes.

In some embodiments, the note recognition system 600 includes a noteauthentication module 650 configured to authenticate the plurality ofnotes. In some cases, at least some of the plurality of notes have amark. The mark can be used to facilitate the recognition, extraction andauthentication of the note. For example, the note authentication module650 can obtain an authentication signal from a mark of a note and verifythe authentication based on the authentication signal. In some cases,the note authentication module 650 is coupled to the note recognitionmodule 630 and provides the authentication information to the noterecognition module 630. In some embodiments, the note recognition module630 and the note extraction module 640 can perform the steps illustratedin FIG. 6B and in the relevant discussions.

In some embodiments, a note recognition/management system may usemultiple recognition algorithms to recognize notes and extract notes'content, such as color recognition, shape recognition, and patternrecognition. For example, the system may use color spaces such as theRGB, HSV, CIELAB, etc. to identify regions of interest corresponding tothe notes for color recognition. In some cases, the notes are furtherdistinguished in their shape and due to the presence of unique patternsdetected by shape recognition (e.g., Hough transform, shape context,etc.) and pattern recognition algorithms (e.g., Support Vector Machine,cross-correlation, template matching, etc.) respectively. Thesealgorithms help filter out unwanted objects in the visual representationor other sources of notes' content and leave only those regions ofinterest corresponding to the notes.

In some embodiments, a note may include a mark made using fluorescentmaterials, such as printable inks or coatable dyes. For example, afiducial mark such as a logo can be printed in fluorescent ink on thewriting surface of a note. An appropriate light source would excite thefluorescent material. For example, a white LED (light emitting diode) ofa mobile handheld device may be able to excite the fluorophore using thesignificant blue wavelength component of the LED output spectrum. In oneembodiment, a fluorescent dye can be coated on the writing surface ofthe notes or included in the materials making the notes. In thisembodiment, the fluorescent dye can provide not only verification of abranded product, but can also improve the contrast between the writteninformation on the note and the background of the note itself. Such amark can facilitate the recognition and segmentation of notes on avisual representation produced by an image sensor. In the cases of notesmade from paper with fluorescent dye, the captured visual representationmay have better readability. Depending on the excitation wavelengthrange and the fluorescing wavelength range for the fluorescentmaterials, additional optical equipment, such as a filter, may be usedtogether with an image sensor (e.g., a camera) to improve detection.

FIG. 10A illustrates a flow diagram of an embodiment of a notemanagement system. First, the system receives a plurality of notes frommultiple sources (step 710A). For example, the note management systemmay receive a set of images of a number of notes from a camera or asmart phone and receive another set of images of a number of notes takenfrom a remote location. As another example, the note management systemmay receive a visual representation (e.g., a video) of a number of notestaken by a video recording device and a text stream of notes that isentered via a laptop. In some embodiments, the multiple sources aresources of notes' content taken from different devices, for example,cameras, scanners, computers, etc. Then, the system recognizes one ofthe plurality of notes (step 720A). The system extracts content of theplurality of notes (step 730A). In some embodiments, some notes includemarks (e.g., color block, color code, barcode, etc.) on the note and onesource of notes is a visual representation of some of the notes. In somecases, the recognizing step includes recognizing marks on the notes fromthe visual representation and then determines the general boundaries ofthe notes based on the recognized marks. In some of these cases, theextracting step includes extracting the content based upon therecognized marks, known shapes of the notes, and known relativepositions of the marks on the notes. After the content of the pluralityof notes is extracted, in some cases, the system may associate each ofthe plurality of notes with a unique identifier (step 740A). The systemmay label each of the plurality of notes with a category (step 750A).The labeling step is discussed in more details below. Additionally, thesystem may first authenticate the notes before recognizing the notes.Optionally, the system may present the content of the plurality of noteson a display (step 760A). In some embodiments, the system may presentthe extracted content of the plurality of notes with an output fieldindicating the categories of the notes. In some implementations, thesystem may use the category information of the notes in presenting thenotes (e.g., show the notes in one category in a group).

FIG. 10B illustrates examples of how a system may label a note. In oneembodiment, a note management system can label a note based on adetected signal (710B). For example, a note can have a mark including abarcode; the system can read the barcode and label the note based on thebarcode. In some cases, the system can label a note based on arecognized mark (720B). For example, the mark can include an icon, logo,a colored block, or other graphical symbol indicating a particulargroup. The system may further label a note based on its content (730B).In some cases, the system may label a noted based on historical dataand/or predictive modeling (740B). In some other cases, the system maylabel a note by user input (750B). A note management system can use oneor more approaches to label a note. The system may also use otherapproaches to label a note that are not listed in FIG. 10B, for example,label a note based on the shape of the note. Further, in some cases, anote may be associated with more than one category.

FIG. 11 illustrates a module diagram of a note management system 800. Inthe embodiment as illustrated, the system 800 includes one or more notesources 820, a note recognition module 830, a note extraction module840, and a note labeling module 860. Various components of the notemanagement system 800 can be implemented by one or more computingdevices, including but not limited to, circuits, a computer, aprocessor, a processing unit, a microprocessor, and/or a tabletcomputer. In some cases, various components of the note managementsystem 800 can be implemented on a shared computing device.Alternatively, a component of the system 800 can be implemented onmultiple computing devices. In some implementations, various modules andcomponents of the system 800 can be implemented as software, hardware,firmware, or a combination thereof. In some cases, various components ofthe note management system 800 can be implemented in software orfirmware executed by a computing device. Additionally, variouscomponents of the system 800 can communicate via or be coupled to via acommunication interface, for example, a wired or wireless interface. Thenote source(s) 820 is configured to provide a plurality of notes. Insome embodiments, one of the note sources 820 is a visual representationof a scene having one or more notes. In some cases, the note sources 820include a plurality of different sources for providing notes, forexample, images, text stream, video stream, or the like. The noterecognition module 830 is coupled to the note sources 820, the noterecognition module 830 is configured to receive the visualrepresentation and determine a general boundary of a note from thevisual representation. The note extraction module 840 is configured toextract content of the one of the plurality of notes from the visualrepresentation based on the determined general boundary of the one ofthe plurality of notes. The note labeling module 860 is configured tolabel the one of the plurality of notes with a category.

In some embodiments, the note management system 800 includes a noteauthentication module 850 that is configured to authenticate theplurality of notes. In some cases, at least some of the plurality ofnotes have a mark. The mark can be used to facilitate the recognition,extraction and authentication of the note. For example, the noteauthentication module 850 can obtain an authentication signal from amark of a note and verify the authentication based on the authenticationsignal. In some cases, the note authentication module 850 is coupled tothe note recognition module 830 and provides the authenticationinformation to the note recognition module 830. In some embodiments, thenote recognition module 830 and the note extraction module 840 canperform the steps illustrated in FIG. 6B and in the relevantdiscussions. In some embodiments, the note labeling module 860 can useone or more labeling approaches illustrated in FIG. 10B and in therelevant discussions. In some cases, a note may be associated with morethan one category.

In some embodiments, the note management system 800 may include a notepresentation module 870, which is configured to gather content of theplurality of notes and present at least part of the plurality of notesaccording to the category of the notes. For example, the notepresentation module 870 can organize the plurality of notes into groupsthat have the same category. As another example, the note presentationmodule 870 can add connection lines and/or arrows of different groups ofnotes.

FIG. 12A illustrates an example of content extraction process of asegmented note with marks. A visual representation 920A of a note 910Ais captured by an imaging device (e.g., a smart phone 922A). The note910A is segmented into four sections using four set of marks, 912A,914A, 916A, and 918A. Each of the four segments has a correspondingcontent, Content A 950A, Content B 960A, Content C 970A, and Content D980A. The computing device can use any of the note extraction approachto extract the note content and present to a user, as illustrated in930A. In addition, a category can be associated with a specific markhaving a known geometric shape and/or size, as illustrated as 924A.Using the category association, the pieces of content, Content A-D, areassociated with a category as shown in 930A.

FIG. 12B illustrates an example of content extraction process of a note910B with color segments. 920B is a visual representation of the note910B captured by an image sensor, for example, a smart phone 922B. A setof content 940B, each piece of content corresponding to a segment, areextracted using any of the note extraction process as described above.In addition, a piece of content can be associated with a categorydepending on the color of the color segments, and the list of contentpresented to a user can be grouped by the categories, as illustrated in930B.

FIG. 12C illustrates an exemplary scenario of marking a board 910D toextract segmented content. Marks 915D are attached to board 910D. Insome cases, the relative positions (e.g, top left corner, bottom rightcorner) of the marks are known and used to extract content. The piecesof content 920D, 930D, 940D and 950D can be extracted using, forexample, the pseudo code listed in Table 5, where the marks 915D areidentified as bounding boxes.

TABLE 5 Pseudo Code for Extracting Content Use color classificationalgorithm to identify bounding boxes and place in a collection WhileSearch bounding box collection for bounding box with the minimum X value(left most) Search bounding box collection for bounding box that is tothe right and lower than note found in previous step (constraint is thatnote must be at least a defined distance to the right +X) Used boundingboxes to define the region of information to be extracted Discard bothbounding boxes from collection Notes left in collectionThis simple algorithm can be very effective in providing a method forextracting content from free flowing discussions on a board.

FIGS. 13A and 13B illustrate an example of extracting colored notesusing color classification algorithms. FIG. 13A illustrates a visualrepresentation 1000A of notes with different colors, where Yrepresenting yellow, M representing magenta, and B representing blue.FIG. 13B represents extracting all yellow notes 1000B from the visualrepresentation using, for example, the pseudo code listed in Table 2.

The present invention should not be considered limited to the particularexamples and embodiments described above, as such embodiments aredescribed in detail to facilitate explanation of various aspects of theinvention. Rather the present invention should be understood to coverall aspects of the invention, including various modifications,equivalent processes, and alternative devices falling within the spiritand scope of the invention as defined by the appended claims and theirequivalents.

What is claimed is:
 1. A method of extracting content from a note usinga computer system having one or more processors and memories,comprising: capturing, by a sensor, image data comprising a visualrepresentation of a note, wherein the note comprises a physical notehaving one or more segments, each of the segments having at least onemark having a known characteristic; identifying, by the one or moreprocessors, marks within the image data; based on the knowncharacteristics of identified marks, determining, by the one or moreprocessors, a general boundary of the physical note; based on the knowncharacteristics of identified marks and the determined boundary of thephysical note, determining, by the one or more processors, generalboundaries within the image data for the one or more segments of thephysical note; extracting, by the one or more processors and from theimage data, content comprising a set of content pieces using the generalboundaries, each of the content pieces corresponding to a different oneof the one or more segments of the note; creating, by the one or moreprocessors, a digital note representative of the physical note, thedigital note having data indicating each of the segments for thephysical note; and associating, by the one or more processors, therespective content pieces extracted from the image data with thecorresponding segments of the digital note.
 2. The method of claim 1,wherein the marks comprise at least one of lines, arrows, star-shapedmarks, elbow-shaped marks, rectangular marks, circular marks,ellipse-shaped marks, polygon-shaped marks, and geometric-shaped marks.3. The method of claim 1, further comprising: identifying a shape foreach of the one or more marks; comparing the identified shape with a setof known shapes for marks to be affixed to physical notes; andassociating each of the pieces of content extracted from a respectiveone of the one or more segments with a category based on the comparisonof the identified shape of the at least one mark for the correspondingsegment to the set of known shapes.
 4. The method of claim 3, furthercomprising: presenting the one or more of the content pieces on agraphical interface of a computing device with an indicator indicativeof its category.
 5. The method of claim 3, further comprising:presenting the content pieces on a graphical interface of a computingdevice in one or more groups, wherein each of the content pieces in asame one of the groups is associated with a same category.
 6. The methodof claim 1, further comprising: identifying a color for each of the oneor more marks; and associating each of the pieces of content extractedfrom a respective one of the one or more segments with a category basedon the identified color of the at least one mark for the correspondingsegment.
 7. The method of claim 6, further comprising: presenting theone or more of the content pieces on a graphical interface of acomputing device with an indicator indicative of its category.
 8. Themethod of claim 6, further comprising: presenting the content pieces ona graphical interface of a computing device in one or more groups,wherein each of the content pieces in a same one of the groups isassociated with a same category.
 9. The method of claim 1, wherein thedetermining step comprises determining the general boundaries based uponthe identified marks and known relative positions of the marks for theone or more segments.
 10. A note management system having one or moreprocessors and memories, comprising: a sensor configured to capture animage data of a physical note, wherein the note is separated into one ormore segments using marks, wherein each of the segments is marked by atleast one of the marks having a known characteristic, a note recognitionmodule coupled to the sensor, the note recognition module executing onthe one or more processor and configured to receive the captured imagedata and identify the marks within the image data; and a note extractionmodule executing on the one or more processors and configured todetermine a general boundary of the physical note based on the knowncharacteristics of identified marks and determine general boundaries ofthe one or more segments within the captured image data based on theidentified marks and the determined boundary of the physical note andextract content using the general boundaries of the one or moresegments, the content comprises content pieces, each of the contentpieces corresponding to one of the one or more segments of the note,wherein the note extraction module is further configured to create adigital note representative of the physical note, the digital notehaving data indicating each of the segments for the physical note, andassociate the respective content pieces extracted from the image datawith the corresponding segments of the digital note.
 11. The notemanagement system of claim 10, wherein the note extraction module isfurther configured to determine the general boundaries of the one ormore note segments based upon the identified marks and predefinedrelative positions of the marks for the one or more note segments. 12.The note management system of claim 10, further comprising: a notepresentation module executing on the one or more processors andconfigured to present the set of content on a user interface.
 13. Thenote management system of claim 12, wherein the note presentation modulepresents at least one of the content pieces with an indicator indicativeits category.
 14. The note management system of claim 12, wherein thenote presentation module is further configured to present the contentpieces in one or more groups, wherein each of the content pieces a groupis associated with a same category.
 15. The note management system ofclaim 10, wherein the marks comprise at least one of lines, arrows,star-shaped marks, elbow-shaped marks, rectangular marks, circularmarks, ellipse-shaped marks, polygon-shaped marks, and geometric-shapedmarks.
 16. The note management system of claim 10, wherein the noteextraction module identifies a shape for each of the one or more marksand associates each of the content pieces with a category by the shapeof the at least one mark for the corresponding segment.
 17. The notemanagement system of claim 10, wherein the note extraction moduleidentifies a color for each of the one or more marks and associates eachof the content pieces with a category identified by the color of the atleast one mark for the corresponding segment.
 18. The note managementsystem of claim 12, further comprising: a database having a plurality ofrecords storing data indicative of a plurality of digital notes, whereinthe note extraction module creates a digital note within the databaserepresentative of the physical note captured by the image data, thedigital note stored within the database and having data indicating eachof the segments identified for the physical note, and wherein the noteextraction module updates the records to associate the respectivecontent pieces extracted from the image data with the correspondingcolor segments of the digital note.
 19. A non-transitorycomputer-readable storage device comprising instructions that cause aprogrammable processor to: receive image data containing a visualrepresentation of a scene having a plurality of physical notes, each ofthe plurality of physical notes having a plurality of segments, each ofthe segments having at least one mark having a known characteristic;process the image data to identify the marks within the image data anddetermine, based on the known characteristics of the marks, a boundarywithin the image data of a first physical note of the plurality ofphysical notes; process the image data to determine, based on the knowncharacteristic of the marks and the boundary determined for the physicalnote, boundaries for each of the segments of the first physical note;extract, from the image data, respective content for each of thesegments of the first physical note using and the boundaries determinedfor each of the segments of the physical note; create a digital noterepresentative of the first physical note, the digital note having dataindicating each of the segments for the physical note; and associate therespective content pieces extracted from the image data with thecorresponding segments of the digital note.